feat(pi): deploy from CI-gated ref within a minute (candidate A: gated pull)#483
Conversation
Pi converged on main only nightly. Now CI fast-forwards a `deploy` ref after `all-builds` passes, and pi's autoUpgrade polls it every minute, gated by an ExecCondition so idle ticks cost one `git ls-remote` instead of a 3-5 min flake eval on the Pi 4. The gate records the rev it acted on before switching and promotes it only on success, so a rev pushed mid-deploy is caught next tick, never lost. Gating on `deploy` (never raw main) also guarantees the aarch64 closure is already in Cachix, pushed by tower's post-build hook in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| case "''${1:-}" in | ||
| check) | ||
| mkdir -p "$state_dir" | ||
| rev=$(git ls-remote "$deploy_url" deploy | cut --fields=1) || true |
There was a problem hiding this comment.
no || true, if it fails it fails
There was a problem hiding this comment.
Done, removed. Consequence checked: a network failure makes git ls-remote exit 128, which set -e -o pipefail propagates as the script's exit code — and per man systemd.service, ExecCondition exit codes 1–254 mean "condition failed, skip the run" without marking the unit failed (only 255/abnormal would). The no-deploy-ref-yet case is separate: ls-remote exits 0 with empty output, so the [ -z "$rev" ] guard still skips cleanly.
There was a problem hiding this comment.
Do you think we should make it fail the unit?
There was a problem hiding this comment.
I'd say no. At one tick per minute, transient network failures are routine — a failed unit on every blip would light up systemctl --failed (and any future OnFailure hooks) with nothing for you to act on, while the skip already gives the right behavior: the next tick retries automatically. Failure states earn their place when someone must intervene; a poll error doesn't qualify. If prolonged-outage visibility ever matters, an OnFailure ntfy on the deploy itself (or a monitoring-side staleness check) is the better tool than failing the poll.
There was a problem hiding this comment.
Can we add a notification on failure using ntfy-wrapped or something?
There was a problem hiding this comment.
Already covered, it turns out — no new code needed. modules/ntfy-failure-alerts.nix (#408, in nixosBase, which pi imports) ships a type-level systemd drop-in adding OnFailure=ntfy-failure@%n.service to every system service, nixos-upgrade included; the template posts the failing unit's last journal lines via ntfy-wrapped titled "nixos-upgrade.service failed on pi". Semantics are exactly what we wanted: ExecCondition skips (poll unchanged, network blip) don't mark the unit failed so they never notify, while a real failed deploy does. Adding a dedicated unit here would be pure redundancy — systemd already resolves the drop-in to OnFailure=ntfy-failure@nixos-upgrade.service.
| echo "deploy $rev already live; skipping" | ||
| exit 1 | ||
| fi | ||
| printf '%s\n' "$rev" > "$pending_rev" |
There was a problem hiding this comment.
question: isn't that the same as echo $rev > "$pending_rev"?
There was a problem hiding this comment.
Effectively yes — for a git SHA there are no escape sequences or leading dashes where they'd differ. Simplified to echo "$rev" > "$pending_rev".
There was a problem hiding this comment.
Do you think it is good practice to use printf because sometimes there might be escape sequences or leading dashes?
There was a problem hiding this comment.
As a general habit, yes — POSIX leaves echo's handling of backslashes and option-like arguments implementation-defined (dash and zsh interpret escapes; a value starting with - can be eaten as a flag, and echo has no --), so printf '%s\n' is the robust default for arbitrary data. Here neither hazard can occur: bash's echo (writeShellApplication pins bash) doesn't expand escapes by default, and a git SHA is [0-9a-f]{40} — can't start with - or contain backslashes. So echo is safe in this spot; happy to switch back to printf if you'd rather apply the habit uniformly.
There was a problem hiding this comment.
Done — switched back to printf '%s\n' "$rev" > "$pending_rev".
- Drop the auto-upgrade sentence from the LAN DHCP+DNS note in CLAUDE.md
- Let systemd own the state dir via StateDirectory; drop mkdir -p
- Simplify the deploy gate script (no ${1:-}, no `|| true`, plain mv/echo)
- Document the every-minute autoUpgrade cadence inline
- Drop redundant needs.all-builds.result check and stale comments
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move the deploy-gate script, system.autoUpgrade settings, and the nixos-upgrade ExecCondition/ExecStartPost/StateDirectory wiring out of pi's configuration.nix into modules/gated-upgrade.nix, registered as flake.nixosModules.gatedUpgrade and imported only by pi. Built system is unchanged (identical toplevel drvPath). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
Closes #481
Candidate A (gated pull) of the two implementations #481 calls for — to be compared against candidate B (#482); the loser doesn't merge.
What
deployCI job (main pushes only, afterall-buildssucceeds,contents: write) fast-forwards thedeployref:git push origin HEAD:deploy.system.autoUpgraderetargetsmain→deploy, ticks every minute (dates = "*:0/1", norandomizedDelaySec).ExecConditiongate script (pi-deploy-gate check) makes idle ticks cost a singlegit ls-remoteagainst a last-deployed-rev state file — no on-pi flake eval unlessdeployactually advanced.ExecStartPost(pi-deploy-gate record) promotes the gated rev only after a successful switch.Invariant: never deploy raw main
deployonly moves whenall-buildsis green, which also guarantees the aarch64 closure is already in Cachix (tower's post-build hook pushes during the CI build).Race handling
checkrecords the rev it gated on topending-rev;recordpromotes it tolast-deployed-revonly on ExecStart success. A rev pushed mid-deploy causes at most one redundant (cache-hit) redeploy on the next tick, never a skipped rev. Failed deploys leave the state untouched, so the next tick retries. Missingdeployref or network failure → clean skip, not a unit failure.Comparison criteria (vs B)
🤖 Generated with Claude Code